I believe the spark version bump switched the runtime logging implementation to log4j core v2. This broke our py4j logging configuration tacitly. After switching over to v2 I discovered that we don't need things like the StringSocketAppender​ because we're redirecting the jvm's stderr to python.

We use the hail jar in a few situations with slightly different requirements - tests, py4j, py4j in batch and qob:

• For qob we want log4j to append to stdout as that'll appear in the job logs.
• For tests, we want low verbosity lest we're driven to madness searching for failure logs in the test output
• For py4j in batch, we want the job log to contain what we usually log to the hail log file as that's lost after a job terminated.
• For py4j in an ipython session, we probably only want warnings or errors or our users will hate us.

This is what Py4jUtils.configureLogging​ attempts to address: when binging to apy4j​ gateway, we need to reconfigure the logging configuration so we can forward logs to stderr and write to our logfile.

Note that the configuration contains a spark logger - the logs are quite noisy and adding this allows us to control how much noise we want from spark.

This change cannot impact the Hail Batch instance as deployed by Broad Institute in GCP